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Sadly, scandal has once again brought attention to the nation’s child welfare system. In 
response to the news that Florida’s child welfare bureaucracy could not account for 
hundreds of children supposedly in its care, investigative journalists have found evidence of 
similar problems in other states (Anderson, 2002; Kresnak, 2002; News- Journal Wire 
Services, 2002). Ironically, the news that some states cannot even locate children comes at a 
time when the federal government is in the midst of trying, for the first time, to hold states 
accountable for achieving quantifiable outcomes for children involved with the child 
welfare system. These outcomes focus on the safety, permanency and stability of living 
arrangements, and well-being of children. The juxtaposition of high hopes for improving 
state accountability for foster children and the continuing legacy of poor performance by 
child welfare bureaucracies calls for a sober assessment of current federal efforts to help 
states improve outcomes. 

In this paper we critique the national standards used in the new Child and Family Service 
Reviews being conducted by the federal government and suggest new directions. First, we 
provide a historical perspective on current federal policy and a description of the new 
performance measurement system. We then point out the conceptual limitations of the 
current national standards and use empirical evidence to illustrate some of these limitations. 
We conclude with recommendations for new standards and additional efforts that will be 
necessary to ensure that state performance can improve. 
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INTRODUCTION 

Sadly, scandal has once again brought attention to the nation’s child welfare 
system. In response to the news that Florida’s child welfare bureaucracy could not 
account for hundreds of children supposedly in its care, investigative journalists have 
found evidence of similar problems in other states (Anderson, 2002; Kresnak, 2002; 
News-Journal Wire Services, 2002). Ironically, the news that some states cannot even 
locate children comes at a time when the federal government is in the midst of trying, for 
the first time, to hold states accountable for achieving quantifiable outcomes for children 
involved with the child welfare system. These outcomes focus on the safety, permanency 
and stability of living arrangements, and well-being of children. The juxtaposition of 
high hopes for improving state accountability for foster children and the continuing 
legacy of poor performance by child welfare bureaucracies calls for a sober assessment of 
current federal efforts to help states improve outcomes. 

In this paper we critique the national standards used in the new Child and Family 
Service Reviews being conducted by the federal government and suggest new directions. 
First, we provide a historical perspective on current federal policy and a description of 
the new performance measurement system. We then point out the conceptual limitations 
of the current national standards and use empirical evidence to illustrate some of these 
limitations. We conclude with recommendations for new standards and additional efforts 
that will be necessary to ensure that state performance can improve. 
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BACKGROUND 

As early as the 1930’s, there were those who advocated for the use of rigorous 
research methods to assess social programs (Rossi & Freeman, 1982). In the post WWII 
era, the implementation of many new, large-scale, and costly programs in the health and 
human services arena were accompanied by demands that these programs be evaluated. 
By the end of the 1950’s, such evaluations were commonplace. Improvement in data 
collection techniques and the advent of computer technology further enhanced outcome 
evaluation techniques. With the tremendous expansion of social programs during the 
1960’s, program evaluation became an integral part of policymaking, planning and 
administration — a “growth industry” (Rossi & Freeman, 1982, p.22). In the early 1970’s, 
however, economic recession led to a reappraisal of the effectiveness of many Great 
Society programs. Declining support for public services led to significant funding and 
program cuts and increased calls for accountability. Since then, performance standards 
and outcome monitoring have become critical to program management as both public and 
private funding sources demand greater accountability. Most recently. Congress passed 
the Government Performance Results Act of 1993 (GPRA) which required all federal 
agencies to work with the states to establish performance goals and monitor performance 
results for all federal programs (Office of Management and Budget, 2002). 

This growing emphasis on accountability combined with the advent of the micro- 
computer helped facilitate the development of administrative databases for program 
management and evaluation purposes (Magura & Moses, 1986). The benefits (and 
challenges) of working with administrative data in child welfare have been well 
documented (Bom, 1 997, Courtney & Collins, 1 994, Drake & Jonson-Reid, 1 999, 
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English, Brandford, & Coughlan, 2000, Goerge, 1997). Over time, administrative data 
has taken on an increasingly important role in the evaluation of child welfare services. 

From the late 1940’s until 1975 the federal government collected voluntary data 
on foster care and adoption (USDHHS, 2002a). From then until the early 1980’s little or 
no state specific data were reported. Activities emerging from the 1980 Adoption 
Assistance and Child Welfare Services Act (PL 96-272), such as state self-assessment 
and review guidelines for the enforcement of Titles IV-B and IV-E, were the beginning 
of the outcomes movement in child welfare (Children and Family Research Center, 
2000 ). 

In 1982, the American Public Welfare Association (APWA) — currently known as 
the American Public Human Services Association (APHSA), began to publish data from 
their Voluntary Cooperative Information System (VCIS), with fimding assistance from 
the United States Department of Health and Human Services’ Administration for 
Children and Families (ACF)(USDHHS, 2002a). Although VCIS provided the best 
available national statistics about foster care and adoption, the system had significant 
limitations. Data were reported via a survey instrument, filled out by the primary state 
agency administering child welfare programs, and were assumed to be correct. States 
used different reporting periods (e.g., federal fiscal year, state fiscal year, calendar year), 
and some states deviated from the programmatic definition categories for foster care and 
adoption defined by APWA. After making adjustments for internal consistency and 
conducting other data flow edits, APWA used VCIS to calculate annual national 
estimates of the number of children in care. Though these estimates were based on 
responses from a high of 52 states in 1982, by 1986 data from only 32 states were 
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available. In 1986, Congress amended Title IV-E of the Social Security Act by adding 
section 479, which provided the directives for establishing and implementing a 
mandatory foster care and adoption reporting system. 

It was not until 1 994 that funding became available to help states implement their 
Adoption and Foster Care Analysis and Reporting Systems (AFCARS). The Omnibus 
Budget Reconciliation Act of 1993 (PL 103-66) provided enhanced funding for Statewide 
Automated Child Welfare Information Systems (SACWIS) (Federal Register, 1993a and 
1993b). States could choose a stand-alone AFCARS system for which the federal 
government would match 50 percent costs or receive a 75 percent federal match for a 
SACWIS that integrated AFCARS requirements into a larger comprehensive system that 
also included: systems operated under the Title IV-A program (AFDC/TANF), the 
National Child Abuse and Neglect Data System (NCANDS)*, systems operated under 
Title XIX (Medicaid), as well as systems operated under Title IV-D (child support 
enforcement) (Collins, 1999)). Most states have opted for the SACWIS alternative.^ 

States began AFCARS data collection in 1994 with first submissions due in 1995. 

The Adoption and Safe Families Act of 1997 (P.L. 105-89) helped further 
promote accountability in child welfare. Section 203 of ASFA directs USDHHS to 
develop a set of outcome measures (including length of stay in foster care, number of 
foster care placements, and number of adoptions) that can be used to assess the 

' The Child Abuse Prevention and Treatment Act (Public Law 93-247), as amended, called for the 
establishment of a national data collection and analysis program on child abuse and neglect. In response, 
USDHHS established NCANDS as a voluntary national reporting system for states. 

^ As of September 1, 2001, 4 states had completed SACWIS systems, 21 were operational, 8 partially 
operational, 8 were in the implementation stage, 6 in the planning stage, and 4 had no activity. (USDHHS, 
2001) 
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performance of states in operating child protection and child welfare programs. AFSA 
requires that, to the extent possible, the outcome measures should be developed from 
AFCARS, and mandates the preparation and submission of an Annual Report to 
Congress regarding state-specific performance on each outcome measure. (USDHHS, 
1999)^. A Final Rule"* * effective March, 2000, among other things, established the Child 
and Family Services State Reviews (CFSRs). The CFSRs assess state outcomes in three 
primary domains; 

Safety 

• Children are, first and foremost, protected from abuse and neglect. 

• Children are safely maintained in their homes whenever possible and appropriate. 
Permanency 

• Children have permanency and stability in their living situations. 

• The continuity of family relationships and connections is preserved for children. 

Family and Child Well-Being 

• Families have enhanced capacity to provide for their children’s needs. 

• Children receive appropriate services to meet their educational needs. 

• Children receive adequate services to meet their physical and mental health needs. 

^ The most recent Annual Report available is for 1999 and can be found online at 
http://www.acfhhs.gov/programs/cb/publications/cwo99/outcomes.pdf 

• htt p://www.acfdhhs.gov/programs/cb/laws/fed reg/frO 12500.htm 
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Each CFSR is a two-part process that includes a statewide assessment and an 
onsite review of these seven child outcomes and seven program systemic factors. ^ For 
the statewide assessment, the Children’s Bureau prepares and transmits to the State the 
data profiles that contain aggregate data on the State’s foster care and in home service 
populations. The data profiles allow each State to compare certain safety and permanency 
data indicators with national standards determined by the Children's Bureau. 

The national standards (which are a subset of the outcomes in the Annual Reports 
to Congress) are based on information that is reported by States to the Detailed Case Data 
Component of the National Child Abuse and Neglect Data System (NCANDS) and the 
Adoption and Foster Care Analysis and Reporting System (AFCARS). Most states 
submit a record for each report alleging child abuse or neglect that received a disposition 
as a result of an investigation or an assessment during the calendar year to NCANDS 
(USDHHS, 2002b). Data submitted to AFCARS for any federal fiscal year contains data 
about all children for whom the state IV-E/IV-B agency has responsibility for placement, 
care or supervision (Collins, 1999). Neither database links files for children from year to 
year; each year ’s database is essentially independent, containing information only about 
those children who experienced the event of interest (i.e., maltreatment disposition or 
foster care placement) during the year. Currently, six statewide data indicators under 
the outcome domains of safety and permanency are being assessed as part of the review 
process. These indicators are: recurrence of maltreatment, incidence of child abuse 
and/or neglect in foster care, foster care re-entries, length of time to achieve reunification. 




’ The onsite reviews began in March 2001, and will continue through 2004. See 

http://www.acf.dhhs.aov/Droerams/cb/cwrD/geninfo/staterev.htm for schedule. The seven systemic factors 
include: statewide information system, case review system, quality assurance system, staff training, service 
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length of time to achieve adoption, and stability of foster care placement. ^ National 
Standards were set at approximately the point where 25% of the states who reported data 
had better and 75% had worse outcomes,^ in recent NCANDS (1997-1998 calendar year) 
and AFCARS (1998 fiscal year) submissions, and are as follows: 

Recurrence of maltreatment —National Standard 6.1% or less: 

Of all children who were victims of substantiated or indicated child abuse and/or neglect 
during the first six months of the reporting period, what percent had another substantiated 
or indicated report within a six-month period? 

Incidence of child abuse and/or neglect in foster care— National Standard 0.57% or 
less: 

Of all children in foster care during the period under review, what percent were the 
subject of substantiated or indicated maltreatment by a foster parent or facility staff? 

Stability of foster care placement— A/hrio«a/ Standard 86. 7% or more: 

Of all children who have been in foster care less than 12 months from the time of the 
latest removal, what percent had no more than two placement settings? 



array, agency responsiveness to the community, foster and adoptive parent licensing, and recruitment and 
retention. 

^ Length of stay in foster care, operationally defined as the median length of stay for a first time entry 
cohort, was deleted from the planned indicators because many states would not have achieved this data 
point for the children under review in time. 

" See http://www.acfdhhs.gOv/programs/cb/hotissues/background.htm#six for more detailed information 
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Length of time to achieve reunification— A^arfowa/ Standard 76.2% or more: 

Of all children who were reunified with their parents or caretakers at the time of the 
discharge from foster care, what percent were reunified in less than 12 months from the 
time of the latest removal from home? 

Length of time to achieve adoption— A^ariowa/ Standard 32. 0% or more: 

Of all the children who exited foster care during the period imder review to a finalized 
adoption, what percent exited care in less than 24 months from the time of the latest 
removal from home? 

Foster care re-entries— A/ariowa/ Standard 8. 6%> or less: 

Of all the children who entered care during the year under review, what percent re- 
entered foster care within 12 months of a prior foster care episode. 

ISSUES WITH THE CURRENT MEASURES AND APPROACH 

Having reviewed their origins, we now describe some of the issues with the 
current federal performance measures. First, we describe some inherent problems with 
the use of the cross-sectional and exit cohort data that are the basis of most of the federal 
measures of state performance. Second, we use data from the Multistate Foster Care 
Data Archive to examine the federal outcome measures for (1) length of time to achieve 
reunifications and (2) length of time to achieve adoption. Third, we describe some 
additional concerns with the current federal approach. 
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If the purpose of measuring outcomes is to provide information that can be used 
to examine changes in state performance over time, then performance measures should 
not be based on data from point-in-time cross-sections of the population or from data on 
exit cohorts. Both of these data sources can be biased in the sense that children entering 
care do not have an equal probability of appearing in the observed sample. Cross- 
sectional samples (e.g., all children in care at a particular point in time) tend to be biased 
towards those children who have relatively long stays (Barth et al., 1994). For example, 
a point-in-time sample of all children in care at the end of a given year will include many 
children from earlier annual entry cohorts who stayed more than one year, all entrants 
from the current year that remained in care for at least one year, but none of the short 
stayers from previous years and only some from the current year. In contrast, samples 
from exit cohorts tend to be biased towards those children from recent entry cohorts with 
relatively short stays in care since those with longer stays have not yet exited care. This 
leaves the impression that length of stay is much shorter than it is when measured using 
the experiences of all the children placed in foster care over time. Bias in outcome 
measures presents at least two problems for the federal performance improvement 
system. First, bias makes it impossible for entities whose performance is being assessed, 
in this case states, to accurately measure their success in improving performance over 
time. In fact, state actions based on biased insights could adversely affect outcomes 
despite opposite intentions. Second, bias creates an unfair basis for the federal 
government to determine which states must embark upon performance improvement 
plans and upon which states to assess financial penalties. 
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To the extent possible, state performance should be evaluated using data that best 
reflect the experiences of all the children served. For analyzing performance, it is in most 
instances best to group children who entered out-of-home care during a specific year, 
following their progress through placement until they experience some form of exit. This 
is the so-called entry cohort method combined with a longitudinal perspective. Because 
entry cohorts include all children, the inherent biases are far more limited and the data are 
more easily interpreted, especially if officials are trying to track change over time. 

In the current plan, HHS proposes to compare states on these two indicators at 
two different points in time to determine whether the observed changes are consistent 
with better performance (i.e., more effective policies and practices). The idea is that, 
over time, states should be able to increase the percentage of children whose placement 
experiences meet the standard. To derive the estimate for reunification and adoption, the 
federal approach says that the standard should be applied to all children who exit foster 
care in a given year. 

To illustrate the limitations of the current federal standards, we examined 
performance regarding the length of time to achieve reunification and adoption in six 
states using the Multistate Foster Care Data Archive. This archive is maintained at the 
Chapin Hall Center for Children at the University of Chicago and currently has data on 
1.2 million children in twelve states. The period of our analysis stretches from 1990 to 
1999 and provides an unprecedented look at state performance through the federal lens. 
For this period we developed two measures of state performance. First, we calculated the 
federal performance measures for exit cohorts in each state in each year for the 
reunification and adoption measures. Second, we created measures for both 
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reunification and adoption based on entry cohorts. Specifically, for reunification we 
calculated the percentage of children who entered in each year who were reunified within 
twelve months of entry. For adoption, we calculated for each entry cohort the percentage 
of children adopted within twenty-four months of entry. 



Table 1 shows the performance of six states for the 1990-1999 exit cohorts on the 
current federal performance measure for family reunification. Thus, for each year the 
table shows the percentage of children who exited care in that year that did so within 
twelve months of their most recent removal from home. Table 2 shows state 
performance in reimifying children with their families using entry cohort data. 
Specifically, the table shows the percentage of children entering care in each year that 
were reunified within twelve months of removal from home. 



Table 1 

Of Exits to Reunification, Percent that Exited Within 12 Months 
of Entry to Care by State and Exit Year 



Exit Y ear 


A 


B 


C 


State 

D 


E 


F 


1990 


67.1% 


64.6% 


74.2% 


68.6% 


61.1% 


76.7% 


1991 


60.4% 


62.2% 


70.9% 


64.8% 


49.6% 


74.5% 


1992 


61.7% 


59.9% 


72.9% 


64.4% 


45.0% 


70.5% 


1993 


60.5% 


59.6% 


70.6% 


62.6% 


48.4% 


65.6% 


1994 


59.1% 


66.3% 


67.6% 


66.2% 


48.9% 


70.8% 


1995. 


57.5% 


62.4% 


67.7% 


66.4% 


47.1% 


72.7% 


1996 


46.4% 


60.5% 


67.0% 


67.4% 


56.0% 


73.5% 


1997 


41.9% 


61.6% 


65.3% 


69.0% 


59.3% 


72.1% 


1998 


36.9% 


65.9% 


60.3% 


67.5% 


58.3% 


70.5% 


1999 


39.4% 


65.4% 


59.9% 


68.3% 


54.6% 


73.9% 
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Table 2 

Of All Children Entering Care, Percent Exiting to Reunification Within 12 Months 

by State and Entry Year 
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Entry Year 


A 


B 


C 


State 

D 


E 


F 


1990 


26.3% 


29.0% 


32.2% 


41.3% 


24.8% 


42.6% 


1991 


20.7% 


29.1% 


31.9% 


40.1% 


23.4% 


41.5% 


1992 


17.9% 


28.6% 


33.7% 


40.7% 


23.6% 


42.9% 


1993 


14.9% 


28.2% 


33.3% 


38.6% 


23.6% 


46.3% 


1994 


14.1% 


26.5% 


30.4% 


38.7% 


23.1% 


45.4% 


1995 


15.5% 


27.4% 


31.3% 


38.1% 


23.3% 


43.2% 


1996 


15.1% 


23.8% 


27.9% 


38.0% 


23.7% 


37.6% 


1997 


16.6% 


20.6% 


18.3% 


37.3% 


24.7% 


37.9% 


1998 


15.7% 


23.4% 


14.9% 


37.6% 


24.3% 


34.7% 


1999* 


10.2% 


14.6% 


9.3% 


24.9% 


16.8% 


21.9% 



* Data current through 12/31/1999. Data for 1999 are incomplete. 



Table 3 

Of Exits to Adoption Percent Occurring Within 24 Months of 
Entry to Care by State and Exit Year 



Exit Year 


A 


B 


C 


D 


E 


F 


1990 


27.8% 


39.9% 


37.5% 


15.2% 


13.2% 


35.4% 


1991 


24.9% 


37.7% 


35.9% 


15.3% 


8.1% 


27.9% 


1992 


20.9% 


37.1% 


33.6% 


14.1% 


7.3% 


25.3% 


1993 


16.6% 


33.8% 


29.2% 


12.5% 


5.7% 


24.2% 


1994 


14.4% 


34.1% 


27.4% 


12.7% 


5.7% 


14.3% 


1995 


11.7% 


41.0% 


27.5% 


14.2% 


5.0% 


14.8% 


1996 


10.3% 


40.6% 


22.9% 


14.6% 


4.7% 


18.7% 


1997 


7.3% 


77.0% 


23.0% 


10.9% 


4.9% 


21.9% 


1998 


7.4% 


59.8% 


24.9% 


15.4% 


4.5% 


21.4% 


1999 


7.0% 


42.8% 


27.0% 


14.6% 


5.3% 


18.2% 
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Table 4 

Of All Children Entering Care, Percent Exiting to Adoption Within 24 Months 
by State and Entry Year 



Entry Year 


A 


B 


C 


State 

F 


G 


H 


1990 


2.2% 


6.4% 


4.7% 


1.7% 


0.7% 


1.5% 


1991 


1.6% 


6.8% 


3.9% 


2.0% 


0.8% 


1.4% 


1992 


1.8% 


7.5% 


3.2% 


1.3% 


0.9% 


1.4% 


1993 


1.4% 


8.5% 


3.4% 


2.2% 


1.0% 


1.2% 


1994 


1.3% 


8.5% 


2.8% 


1.9% 


1.1% 


1.1% 


1995 


1.9% 


7.0% 


3.0% 


1.7% 


1.1% 


1.5% 


1996 


2.6% 


11.3% 


3.5% 


2.0% 


1.0% 


2.4% 


1997 


4.6% 


22.1% 


4.2% 


2.3% 


1.1% 


1.9% 


1998* 


3.2% 


7.9% 


3.5% 


1.2% 


0.6% 


1.3% 


1999* 


0.3% 


1.9% 


0.7% 


0.1% 


0.1% 


0.3% 



* Data current through 12/31/1999. Data for 1998 and 1999 are incomplete. 



Table 3 shows the performance for six states for the 1990-1999 exit cohorts on the 
current federal performance measure for adoption. For each year the table shows the 
percentage of children who exited care in that year that did so within twenty-four months 
of their most recent removal from home. Table 4 shows state performance in reunifying 
children with their families using entry cohort data. Specifically, the table shows the 
percentage of children entering care in each year that were adopted within twelve months 
of removal from home. 

Each state’s data tell a somewhat different story. However, two themes emerge. 
First, exit cohort data significantly overestimate both the proportion of children who are 
reunified within 12 months and the proportion of children adopted within 24 months. 

This is completely consistent with the idea that exit cohorts favor children who have been 
in foster care for shorter periods. For example, of the children reimified in 1990, about 
67 percent had been care for 12 months or fewer prior to discharge. That figure dropped 
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rather dramatically, falling to near 40 percent in 1998, suggesting a period when state 
performance was falling behind. In any event, a figure of 40 percent is considerably 
higher than the comparable entry cohort figure. This same pattern characterizes the 
adoption data. 

Second, performance trends - changes in outcomes over time - are sensitive to 
whether one considers only children who are exiting care or if one chooses to look at a 
series of entry cohorts. In fact, if the experience of these six states is any indication, one 
should expect the trends observed for exit cohorts to differ from those observed for entry 
cohorts rather than to be similar. This is true for both reunification and adoption. We use 
state A to illustrate the divergence in outcomes trends between exit and entry cohorts. In 
State A, the entry and exit perspectives provide a similar view of state performance in 
family reunification from 1990 through 1993 (see Figure 1). Performance was trending 
downward. The downward trend continued throughout the 1990s if the data are viewed 
from the exit perspective. However, starting in 1994, the entry data show a slight 
improvement in the proportion of children admitted who went home in 12- months. The 
adoption data are even more striking (see Figure 2). In short, adoption performance 
improved through the mid to late 1990s even though the exit perspective shows continued 
declines. Moreover, the magnitude of performance differences is exaggerated when 
viewed from the exit perspective. Clearly, if the purpose of the federal performance 
measures is to create a meaningful and reliable metric by which to assess state 
performance over time and upon which states can base program planning, then the federal 
government should replace the current measures. 
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Figurel; State A, Exits to Reunification within 12 months of Entry to Care^ as of 12/31/1999 



Percent 
Discharged 
to Reunification 
within 12 
Months 





Notes: 

• Exit - Percent of exits to reunification in a given year occurring within 12 months of entry to care. 
(HHS/CFSR). 

• Entry - Of all children entering care during a given year, percent exiting to reunification within 1 2 
months. 



Figure 2: State A. Exits to Adoption within 24 Months of Entry to Care, as of 12/31/1999 




Notes: 

• Exit - Percent of exits to adoption in a given year occurring within 24 months of entry to care. 
(HHS/CFSR). 

• Entry - Of all children entering care during a given year, percent exiting to adoption within 24 months. 
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We now turn to other limitations of the HHS performance standards, some of 
which can be dealt with through improved use of state administrative data, others that 
will require more serious thought by policymakers about the intent behind the new focus 
on accountability. An obvious limitation of the current performance measures is the fact 
that they are not risk adjusted. In other words, they do not take into account between- 
state differences in the characteristics of the children and families receiving child welfare 
services. For example, observed differences in state performance relative to any of the 
national standards may have more to do with demography than program performance, 
given that children’s age and race, and a state’s urbanicity, are all related to outcomes 
such as reunification and adoption (Wulczyn, Brunner Hislop, & Goerge, 2000). Of 
course, adjusting for risk would not be easy since the relationship between various risk 
factors and many child welfare outcomes is far from clear. Moreover, few state 
administrative databases have data on many potentially important risk factors (e.g., 
children’s prenatal substance exposure, parental housing instability). Nevertheless, at 
present a performance assessment system could be developed that adjusts state 
performance based on basic demographic data that are available in nearly every state’s 
management information system. 

An additional challenge is the fact that changes in performance at one point in the 
child welfare system can have marked effects at other points in the system in ways that 
confoxmd performance measurement. For example, hypothetical State A may be much 
more successful than State B in establishing community-based fairuly supports that 
succeed in diverting children from out-of-home care. In the process. State A may end up 
with a more troubled population than State B in out-of-home care and fare worse relative 
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to out-of-home care performance standards. Ironically, in this scenario the better 
performing state is penalized for its success. Once again, in principle this problem can be 
addressed, if these kinds of system dynamics are understood, by adjusting measures of 
state performance. For example, the measure of state performance in reunifying children 
with their families could be adjusted by a measure of foster care incidence in the state, if 
this were shown to be a good indicator of the challenges to serving the families of 
children in care. In reality, knowledge of child welfare services caseload dynamics is not 
yet at a place to facilitate such calculations. 

A more fundamental but less understood potential problem with the federal 
performance standards is the likelihood that they will lead states to change their policies 
and practices in unintended ways. Given the possible fiscal penalties for failure to meet 
the national standards, states have a strong incentive to alter policy and practice to 
improve any outcome for which they are not meeting the standard. In addition, 
heightened attention to these outcomes by state advocacy groups is likely to put even 
more pressure on states to improve when they are out of compliance. 

This may not sound like such a bad thing, but given how little is known about the 
sensitivity of the outcomes to various administrative practices and the relationship of 
population characteristics to the performance outcomes, there is great potential for 
unintended consequences. For example, one way for states to increase the percentage of 
children reunified with their families within twelve months would be to increase the 
number of children admitted to care for a very short period. There is already great 
between-state variation in the use of short-term shelter care, and, given the current 
reunification outcome measure, this variation favors states with high shelter utilization. 
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However, it seems imlikely that Congress intended states to increase their use of short- 
term foster care in order to “catch up” with states that are meeting the family 
reunification performance measure. 

A similar problem emerges with respect to state choices about who to serve in 
their child welfare systems. For example, some states continue to serve status offenders 
in their foster care systems while others have ceased doing so. Given the reasons that 
these youth enter care (i.e., running away from home and other parent-child problems) it 
seems that they might be more likely to move back and forth between care and home than 
younger children placed primarily because of parental abuse or neglect. They might also 
experience a higher number of placements while in out-of-home care due to their 
behavior problems. Thus, all else being equal, states choosing to admit status offenders 
to care might expect to fare less favorably than other states on some of the current 
performance measures and to face the resulting fiscal penalties. Yet, it is far from clear 
that Congress intended states to stop caring for status offenders because of the new 
performance standards. 



CONCLUSIONS 

In summary, we believe that there is clear and convincing evidence that national 
standards that are a part of the Child and Family Services Reviews should not be based 
on measures derived from cross-sectional or exit cohort data. Performance measures 
based on such data are prone to certain biases. The extent of these biases is well 
understood. Furthermore, we believe that even measures based on entry cohort data will 
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not solve all of the problems facing the laudable effort to increase state accountability for 
serving maltreated children and their families. What should be done to move forward? 

First, the federal outcomes need to include an entry cohort perspective. As part of 
that process, greater attention should be paid to how the measures themselves are 
constructed. This is not a significant problem. For example, state and county child 
welfare agencies in New York, Illinois and California have all developed alternatives, 
based on entry cohort data, to the HHS performance measures. The more difficult 
problem, relatively speaking, has to do with the source of data at the federal level. In 
part, the current measures were adopted because the federal AFCARS data system has a 
limited capacity to produce entry cohort data.* For this reason, AFCARS needs to be 
redesigned. Of course, this means that state AFCARS submissions will change. 

However, it is important to point out that well-tested database designs exist.^ While some 
cost would be involved in a conversion, most modem SACWIS systems can produce 
entry cohort data using a longitudinal framework (i.e., tracking children over time). The 
same is tme of older, legacy systems. Such a shift in measurement strategies would 
vastly improve states’ ability to monitor child welfare programs and execute meaningful 
reform. 

Second, a different process of federal-state interaction around program 
improvement should be developed in order to make the most of better outcome measures 

* The problem also exists in the current method for NCANDS data collection. 

’ Both the Chapin Hall Center for Children at the University of Chicago and the Center for Social Services 
Research at the University of California at Berkeley have worked with states for over ten years in 
developing such database systems. In addition, the Jordan Center for Children and Families and the School 
of Social Work at the University of North Carolina at Chapel Hill have assisted numerous sites with the 
construction of longitudinal databases to track outcomes as a part of the Annie E. Casey Foundation’s 
Family to Family Initiative. 
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while avoiding unintended consequences. Presently, the focus of Child and Family 
Service Reviews is on determining if states are “in substantial conformity” with various 
outcomes and systemic factors. The current national standards are a part of the 
assessment of several of the outcomes regarding safety and permanency, and it is not 
possible to achieve substantial conformity on those outcomes in this first Review cycle 
without meeting the standards. When a state is found to be out of compliance, the focus 
shifts to the state’s plan for achieving substantial conformity. Unfortunately it is hard to 
see how in the current context states can develop a performance improvement plan that 
has a reasonable hope of success. Even putting aside the fact that the current federal 
measures are likely to lead the states astray, the knowledge base regarding the 
relationship between population characteristics, programs and practices, and child 
welfare outcomes is so poor as to offer little sound guidance for program improvement 
(Courtney, 2000). This is an environment that is ripe for quick-fix consultants that have 
nothing worthwhile to sell and administrative gaming strategies that “improve” measured 
outcomes at the cost of serious and unintended consequences. 

A better approach would be for the federal government to see the movement 
towards outcome measurement as an opportunity for the field of child welfare services to 
learn about the relationships between policy, program, and outcome. If fiscal sanctions 
are deemed necessary for political reasons, or because some states simply will not take 
the process seriously without them, then so be it. Nevertheless, the primary focus of the 
Child and Family Service Review process should be to better understand why states are 
achieving particular constellations of outcomes. It is only through the acquisition of such 
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knowledge that child welfare systems will be able to achieve meaningful change in child 
and family outcomes. 

This will require developing better capacity for using data for program evaluation 
and planning within both the federal and state child welfare bureaucracies. Clearly, a 
federal bureaucracy that is unable or unwilling to improve on the flawed performance 
measures that are now in place is ill equipped to move the field forward. Our experience 
tells us that very few states are in a position to do much better. In the past ten years, 
states have vastly improved their data systems, but have not done commensurate work in 
developing the human resources necessary to make use of all the new data. Federal and 
state legislatures must devote resources to helping public child welfare agencies carry out 
their accountability function. States should replicate, perhaps with the help of the federal 
government, promising partnerships between state child welfare agencies and university- 
based researchers. This capacity will not appear overnight, but it will not appear at all 
without a serious and sustained commitment of resources. 
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